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Abstract 

Background: How to assess the impact of research is of growing interest to flinders, policy makers and 
researchers mainly to understand the value of investments and to increase accountability. Broadly speaking the 
term "research impact" refers to the contribution of research activities to achieve desired societal outcomes. The 
aim of this overview is to identify the most common approaches to research impact assessment, categories of 
impact and their respective indicators. 

Methods: We systematically searched the relevant literature (PubMed, The Cochrane Library (1990-2009)) and 
funding agency websites. We included systematic reviews, theoretical and methodological papers, and empirical 
case-studies on how to evaluate research impact. We qualitatively summarised the included reports, as well the 
conceptual frameworks. 

Results: We identified twenty-two reports belonging to four systematic reviews and 14 primary studies. These 
publications reported several theoretical frameworks and methodological approaches (bibliometrics, econometrics, 
ad hoc case studies). The "payback model" emerged as the most frequently used. Five broad categories of impact 
were identified: a) advancing knowledge, b) capacity building, c) informing decision-making, d) health benefits, e) 
broad socio-economic benefits. For each proposed category of impact we summarized a set of indicators whose 
pros and cons are presented and briefly discussed. 

Conclusions: This overview is a comprehensive, yet descriptive, contribution to summarize the conceptual 
framework and taxonomy of an heterogeneous and evolving area of research. A shared and comprehensive 
conceptual framework does not seem to be available yet and its single components (epidemiologic, economic, 
and social) are often valued differently in different models. 
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Background 

It is widely accepted that research is a crucial 
investment to foster innovation, knowledge advance- 
ment, and social and economic development. For 
example a knowledge gain is assumed to result from 
biomedical and basic research; if such an output is then 
properly translated in a short but reasonable time lag, it 
will lead to a better health status for populations and 
patients. Much of the information produced is not easily 
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transferable to patient care and this has led to the con- 
cept of the so called "translational blocks" [1]. Evidence 
produced by applied types of health research, such as 
the "comparative effectiveness" and "health services 
research" elicits its potential impact in a more straight- 
forward way. Health care systems, which are nowadays 
increasingly keen to directly support research, are inter- 
ested in overcoming the translational blocks and to 
facilitate a quicker return of their investment in terms 
of information that would help selecting the more effec- 
tive and cost effective interventions so that quality and 
appropriateness can be maximised [2-4]. 
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By definition, research activities are risky and their 
returns highly unpredictable. So, any attempt to increase 
the research system's effectiveness, and to assure and 
monitor quality, is welcome by the whole scientific com- 
munity and funders [5]. Competition on limited 
resources and different funding modalities also raise 
additional concerns. From the limited available evidence 
on the proportion of investments by research stream, 
funding is skewed toward biomedical and basic research 
which, by definition, require more time to have an 
impact [6]. This has raised a debate between those who 
ask for a priority- setting based on the ability to produce 
relevant, usable, and transferable outputs and those sup- 
porting the view that research should be driven only by 
the researchers' interest. If left only into the close 
boundaries of the "research communities", there is a 
concrete risk that the priority setting becomes self-refer- 
ent and the "bidirectional dialogue" between those that 
generate relevant questions from observation in clinical 
practice and those that are responsible to generate the 
new knowledge remains very limited [7]. Monitoring 
and measuring research impact is a complex objective 
requiring the involvement of many actors within the 
research pipeline. In the past two decades, many theore- 
tical frameworks and methodological approaches to 
measure research impact and returns have been devel- 
oped. The payback [8], the cost-benefit [9,10], and the 
decision making impact models [11] are examples of 
evaluation approaches reported in scientific and health 
policy literature. A partial list with a qualitative descrip- 
tion of the most common frameworks is reported in 
Table 1. All these models share a multidimensional 
approach achieved by categorizing impact and benefit in 
many dimensions and trying to integrate them [12]. A 
set of indicators and metrics are then generally asso- 
ciated to each category of impact. For example, biblio- 
metric indicators (e.g. impact factor) are highly reported 
as a measure of the diffusion and awareness of research 
results. These indicators, though welcome to some 
extent because of their directness, are at best only surro- 
gate measures of impact. Moving toward more robust 
metrics, such as those measuring the health status or 
the economic benefit of a population, is a complex task 
but in some way essential [13]. 

The objective of this overview is to describe the con- 
ceptual and methodological approaches to evaluate the 
impact of biomedical and health research. Specifically, 
we aimed at collecting and qualitatively summarizing 
what is available in the biomedical literature in terms of 
theoretical framework and methodologies, with a speci- 
fic focus on the most valid and reliable indicators of 
impact. Our objective was also to see whether this quali- 
tative analysis would have allowed the identification of a 
preferred model to measure research impact and to 



identify the desirable elements (i.e. dimensions to be 
considered, robust and reliable indicators) that a refer- 
ence model should have. 

Other key elements in the "research governance 
debate" such as the analysis of different modes and 
approaches to research funding, prioritization of topics, 
or the analysis of barriers and facilitators to the transla- 
tion of research results were beyond the scope of this 
overview. 

Methods 

In the context of this overview, the term "research 
impact" refers to any type of output of research activ- 
ities which can be considered a "positive return" for the 
scientific community, health systems, patients, and the 
society in general. We refer to any type of health-related 
research, basic and biomedical, such as new drug or 
technology development - and applied research such as 
clinical trials, health service research, and health tech- 
nology assessment (HTA). 

The complexity and heterogeneity of the topic made 
the conceptualization of this overview much less 
straightforward than typical review on medical interven- 
tions. We therefore followed the methodology recom- 
mended by the Cochrane Collaboration for preparing 
"overviews of review" rather than following the steps 
involved in critically appraising the primary studies for a 
systematic review (SR) [14]. We first searched for SRs 
summarizing theoretical model or methodological 
approaches as well as empirical assessment of health 
research funding programs. To increase the comprehen- 
siveness of our search we also sought primary studies 
(case studies) not included in the selected SRs or pub- 
lished in languages other than English. We included stu- 
dies describing conceptual or methodological 
approaches to evaluate the impact of health research 
programs and the empirical evaluation of specific pro- 
grams, funders, research teams, clinical area, etc. In 
both cases, to be eligible for this review the study should 
have mentioned specific impact categories and the indi- 
cators and metrics used to measure this impact. 

Given the broad perspective of the review, the metho- 
dology to identify relevant studies comprised an iterative 
process. We first performed a systematic search on bib- 
liographic databases (Medline and The Cochrane 
Library) using a modified version of the search strategy 
proposed by Martin Buxton and collaborators [12]. The 
search was limited to the SRs published between 1990 
and 2009. We also tried to include relevant studies and 
reports not included in the eligible SRs (i.e. publications 
in French, Italian, and Spanish) published between 2007 
and 2009. Besides bibliographic databases, we screened 
the research funding and Charity's Foundations websites 
cited in the eligible studies, looking for the grey 
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Table 1 A qualitative description of the most widespread frameworks for the evaluation of research impact 



Model 



Description 



Dimensions of impact 
evaluation 



Main proposed indicators 



Examples and Main 
bibliographic 
references 



Payback 



2. Research 
Impact 



3. Research 
utilization 
ladder 



4. Lavis 
decision- 
making 
impact 
model 



5. Weiss 
Logic Model 



6. HTA 
Organisation 
assessment 
framework 



Organizes together in a sequential and 
systematic way the different aspects on 
the impact of research projects from 
dissemination to potential benefits for 
health care 



Evaluates the influence of research 
results and of the potential concurrent/ 
competing factors (cultural context, 
policy content, decision process) in 
policy making 



Evaluates the ways in which research 
progresses towards its application by 
practitioners and policy makers 



Evaluates the impact on decision 
making of any individual or organisation, 

considering the target audience of 
research and the resources available for 
the assessment 



Analyzes the ratio between input 
(resources), process (activity) and results 
of research (products) 



Effectiveness is measured by the ability 
to impact on decision making 



i) knowledge production; 

ii) research targeting and 

capacity building; 

iii) informing policies and 
product development; 

iv) health benefit; 
v) broader economic benefits. 



i) research related impact 

ii) policy impact 

iii) service impact 

iv) societal impact 



i) transmission (of research 
results to practitioners and policy 

makers) 

ii) cognition (reading and 

understanding) 
iii) reference (quoting of research 
results in reports, studies, 
actions) 

iv) effort (to adopt research 

results) 

v) influence (on choices and 

decisions) 
vi) application 

I) policy makers are the ones 
seeking research (user-puil) 

ii) researchers actively 
disseminating results (producer- 
pull) 

iii) researchers and policy-makers 
are both involved actively 
{exchange measures) 

i) initial benefits 
ii) intermediate benefits 

iii) long-term benefits 



i) productivity 
ii) capacity to attract and 
maintain resources and to 
mobilize external support 
iii) culture and values 
maintenance (independence in 
action, transparency of the 
process, accountability to 
stakeholders) 



i) journal articles, conference 
presentations, books, research 

reports, other disseminative 
material; 

ii) new research lines or know 
how, career promotions, PhD, 

masters; 

iii) guidelines and documents 
addressing policies citations, 

membership of decision panels; 

iv) health outcomes, QALY, 
savings for health care systems 
v) benefits in occupation and 
economic development, 
productivity 

i) knowledge and methodology 
advancements, networking, 
leadership 
ii) establishment of 
collaborations and networks 
iii) evidence-based practice; cost- 
containment, quality of care; 

iv) knowledge, attitudes and 
behaviour, health literacy, socia 
capital, equity, macroeconomy 



Process measures (if limited 
resources are available) 
Intermediate outcome measures 
(by performing surveys) 
Outcome measures (by 
performing cases studies) 



Output: publications 
) awareness of medical research 
results in policy making 
ii) any changes in practice 
iii) any changes in well-being 
and health 

i) volume and productivity of 
outputs, quality, 
comprehensiveness, and 

accessibility 
ii) measure of visibility and 
credibility 



Research programs 
financed by the NHS 
UK [12,38,40,44] 



London School of 
Hygiene and Tropical 
Medicine researchers 
evaluation [45] 



I6| 



Canada 
[11] 



[47] 



Quebec Council on 

Health Care 
Technology, Canada 
[48] 
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Table 1 A qualitative description of the most widespread frameworks for the evaluation of research impact (Continued) 



7. Societa 
Impact 
framework 



8. Balanced 
scorecard 



9. Research 
Assessment 
Exercise 
(RAE) 



1 0. Cost- 
benefit 
Analysis 



Research is considered as the valuation 
of the communication of research 
groups with relevant surroundings 
(industry, general public, scientific 
community, public and policy 
institutions) 

Measures performance and drives 
organizational strategy by incorporating 
organisational aspects together with 
financial performance 

To produce quality profiles for each 
submission of research activity made by 
UK academic institutions 



Research impact evaluated in monetary 
terms 



i) knowledge products 
ii) knowledge exchange and 
esteem 
iii) knowledge use 
iv) attractiveness 

i) financial; 
ii) customers; 
iii) business process; 
iv) learning and growth. 

Three quality profiles are defined 
(panel decides the weight given 
to each profile): 

i) research output (minimum 

50%); 

ii) research context (minimum 

5%); 

iii) other indicators (minimum 

5%). 



i) savings for health care systems 

(direct costs) 

ii) savings for the community on 

the whole (indirect costs) 



i) publications, patents, products 
ii) presentations, consultancies, 
and public lectures 
iii) citations, product use 
iv) further funding 



i) RAE1: staff information 
(volume and type of contracts, 
external collaborators), analysis 
of funding for research fellows; 

ii) RAE2: research output 
(publications, patents, reports, 

database, software, etc); 
iii) RAE3: research scholarships 
iv) RAE4: attractiveness for 
external funding 
v) RAE5a: further information on 
groups of research (networking, 
development of a research 
culture, etc) 

i) QALY 

ii) profits 



Royal Netherlands 
Academy of Arts and 
Sciences 2002 [49,50] 



[51] 

Ontario University 
Health Network [52] 

RAE 2008 [53] 



NIH, USA 
[54] 

National Institute of 
Health 1993 [55] 



literature (i.e. additional reports not indexed in biblio- 
graphic databases). In fact, a large part of the literature 
in this field would be made up of heterogeneous 
publications and critical appraisal reports published by 
the main funding agencies. Details on the search strate- 
gies used and the searched websites are reported in 
Additional file 1. 

Finally, we screened the citations reported in the 
included publications and assessed the literature in the 
field already retrieved by authors. We did not contact 
study or report authors. 

Two reviewers independently selected relevant 
publications by screening titles and abstracts. After the 
retrieval of the selected full text publications and, if 
needed, of associated publications, we extracted the fol- 
lowing details: objective, country and setting, evaluation 
time lag, conceptual model methodology, main results 
and conclusions. 

From the analysis of the literature and with reference 
to the more widely accepted theoretical models we 
attempted a description of the categories of impact 
more frequently measured, focusing on the indicators 
and metrics for each category. 

Given the heterogeneity of study designs, the different 
objectives and the lack of a standard methodology 
between studies, we did not perform a quality assess- 
ment of the methods used in different studies. 



Results 

From the bibliographic databases and web searches we 
identified 1064 records. Among these 38 potentially 
relevant publications were retrieved as full text 
(Figure 1). Sixteen publications were excluded, i.e. 
because descriptive, [15] or dealing with prioritization of 
research topics rather than impact [16,17]. We included 
22 publications, referring to 18 studies: four SRs 
[12,18-21] and 14 primary studies [8,9,22-36]. For each 
study, we synthesized the overall objective, scientific 
area evaluated, country and setting, and the time lag of 
the evaluation. 

Hereafter we briefly presents the four SRs. For details 
please refer to Additional file 2. The included SRs pre- 
sented and discussed comparative analyses of theoretical 
models and empirical evaluations performed in several 
countries from 1990. They all used a public funder's 
viewpoint (e.g. central or regional governments, WHO). 
Besides the evaluation of specific research programs, 
each SR also reported a description of theoretical mod- 
els used as a framework for the assessment and a more 
or less explicit description of categories of impact and 
indicators used in the evaluation. Both the reviews by 
Hanney et al and the one from the Canadian Academy 
of Health Science answered to the broad question "how 
to measure research impact". The first aimed at asses- 
sing how the impact of the UK National Health Service 
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1034 records identified through 
database searching 
(MedLine and The Cochrane Library) 




30 records identified through other sources 
• research funding and evaluation agency websites 
•known literature 





V 



38 full text publications assessed for 
eligibility 


\ 




/ 


22 publications reporting data on 18 
studies included in qualitative 
synthesis 



16 publications excluded 
Main reason for exclusion: 

• descriptive studies 
•prioritarisation of research 
•Non-health topics 



4 systematic reviews 



14 primary studies 



Figure 1 Flow of studies through the different phases of the overview 



Health Technology Assessment program should be 
measured and collected the available models, their 
strengths and weaknesses [12]. The latter was interested 
in defining the impact of the Canadian health research 
and to answer to the broad questions "is there a best 
method to evaluate the impacts of health research in 
Canada?" and "are there best metrics that could be used 
to assess those impacts?" [20]. These SRs included 
studies aimed at describing conceptual or methodologi- 
cal approaches to evaluate the impact of health research 
programs as well as empirical applications of different 
assessment strategies (desk analysis, interview, 
peer-review evaluation, case-studies, etc.) and tools for 
measuring impact (indicators and metrics). The review 
published by Buxton et al in 2004 focused on the esti- 
mation of the economic value of research to society. 
The review reported an analysis of benefits in terms of 
direct cost savings to the healthcare system, commercial 
development, healthy workforce, and intrinsic value to 
society due to the health gain [18]. Lastly, Coryn et al. 
have reported a comparative analysis of 16 national 
models and mechanisms used to evaluate research and 
allocate research funding [21]. 

Hereafter we briefly presents the results of the included 
primary studies. For details please refer to Additional file 
3. The studies covered a broad range of evaluation exer- 
cises sponsored by public and private research funding 
agencies. All studies have been conducted in UK, Austra- 
lia, Canada, and USA, with few exceptions [26,32]. These 
studies were highly heterogeneous in terms of the applied 



theoretical frameworks and methodology. The unit under 
evaluation included researchers (from one single 
researcher to teams and whole institutions) but also med- 
ical discipline (e.g. cardiovascular disease research) or 
type of grants (e.g. from public institutions, charities, 
foundations, etc.). 

The large majority applied a bottom-up evaluation 
approach, where information goes from any "producers" 
of research to any target of research outputs [37]. Two 
studies applied a more strictly econometric approach 
used to estimate return on investments in a top down 
manner [9,36]. The method often used were the desk 
analysis, peer reviewer evaluation, interviews and ques- 
tionnaires to principal investigators or to stakeholders 
with a variety of roles in the research production and 
utilization. 

Across the majority of the SRs and primary studies, 
research impact was assessed alongside several 
dimensions, which can be grouped into five categories: 
"advancing knowledge", "capacity building", "informing 
decision-making", "health benefits", and "broad 
socio-economic benefits". Each category, further split 
into subcategories, had a set of indicators and metrics 
capable of providing the size of the impact (see Table 2). 
The more frequently quoted dimensions of impact were 
advancing knowledge (using bibliometric and citational 
approaches), capacity building (mainly using a desk ana- 
lysis approach), and informing decision-making (through 
the evaluation of how and to what extent research 
findings are included into the decisional processes, 
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Table 2 Description of possible impact categories and relative indicators [adapted from Canadian Academy of Health 
Science [20]) 



Impact 
category 



Proposed Indicators 



Data 
collection 
methodology 



Level of 
application 



Theoretical Models 
quoting this 
category 



Advantages 



Disadvantages 



Advancing ■ Activity (number of peer- Bibliometric Basic, clinical, 
knowledge reviewed publications and citational applied and social 
absolute or relative - e.g. to analysis; desk research, 
the department analysis. Assessment of 

publications); individual 

• Quality (impact factor, researchers, teams, 
relative citation impact, high institution, projects, 

impact publications, funding agencies, 

download numbers); 

• Outreach (co-author 
analysis, field analysis of 

citations); 

• Context and structure 
(relative activity index); 

• Other possible indicators 
(expanded relative citation 
impact, relative download 

rate). 

Capacity • Staff (number of PhD, Desk analysis, 

building Master, researchers, member database and 
of staff); interviews 

• Funding (external sources 

of funding); 

• Infrastructure (grant for 

infrastructure and 
coordination activities); 

• Other possible indicators 
(receptor and absorptive 

capacity) 

Informing • Health care (guidelines and Desk analysis, 
policies and policy documents citations - database and 

product e.g. regional plans, interviews 

development educational material, panel 
representatives) 

• Research (references used 
as background for successful 

funding proposals, 
consulting and support 
activity, curricula citations) 

• Industrial (patents and 
industrial collaboration, 

clustering) 

• Citizens (informative 
packages, dissemination 

activities) 

• Media (journals, radio, tv, 

web) 

Health and • Health (Epidemiologic data, Desk analysis, Clinical and 
health sector incidence, prevalence, database and applied research 
benefits mortality QALYs 1 , PROMs 2 ) interviews Evaluation of 

• Health determinants (risk Case studies, teams, institutions, 
factors, educational and audit projects, funding 
social level of cohesion, agencies 

pollution) 
• System (patient satisfaction, 
waiting lists, compliance and 
adherence to clinica 
guidelines, hospitalization, 
length of inpatient stay, 
adverse effects/ 
complications) 



Basic, clinical, 
applied and social 
research 
Assessment of 
teams, institution, 
projects, funding 

agencies. Not 
recommended at 
individual level. 



Clinical, applied, 
social research 
Assessment of 
individua 
researchers, teams, 
institution, projects, 
funding agencies. 



Payback, research 
impact, research 
utilization ladder, 
Lavis decision-making 

impact model, 
societal impact, RAE 



Payback, research 
impact, research 
utilization ladder 



-wide range of 
applications 

-close to 
research itself 
-objective 
-attributed 
relatively 
straightforwardly 
-accessibility and 
feasibility 
-limited cost 



Payback, research 
impact, research 
utilization ladder, 
Lavis decision-making 
model societal 
impact, RAE 



Payback, research 
impact, research 
utilization ladder, 
Lavis decision-making 
model, 
societal impact 



-wide range of 

applications 
-quite close to 
research itself 
-attributed 
relatively 
straightforwardly 

(partially) 
-accessibility and 
feasibility 
-limited cost 

-Optimal for 
projects funded 
ad hoc to inform 
decision making 
-robustness 
-relevance 
-feasibility 
-limited cost 



-robustness 
-relevance 



-surrogate indicators; 
-not always relevant; 
-not comparable 
across different 
disciplines; 
-robust only if based 
on a sufficient set of 
publications (not fully 

appropriate to 
individual researcher 
evaluation). 



-surrogate indicators; 
-quite subjective; 
-attribution issues; 
-self referenced; 
-non mutually 
exclusive [double 
counting) 



-limited spectrum of 

application 
-time-lag between 
input (research) and 
output (result) 
-quite subjective 
-self referenced 



-many confounders 
-feasibility and cost of 
data collection; 
-time-lag between 
input (research) and 
output (result) 
-possible 
underestimation of 

real impact 
-attribution issue 
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Table 2 Description of possible impact categories and relative indicators [adapted from Canadian Academy of Health 
Science [20]) (Continued) 



Economic • Economic rent (salaries, Desk analysis, Clinical and 
and social employments) database and applied research 

benefits • Licensing returns interviews Evaluation of 

• Product sales revenues Case studies, teams, institutions, 
■ Spin-off companies audit projects, funding 

• Health benefit (QALY and Econometrics agencies 
PROM per health care dollar) 
• Well-being (happiness, level 
of social isolation) 

• Social benefits (socio- 

economic social) 



Payback, societa 
impact, cost-benefit. 



-robustness 
-relevance 



-many confounders 
-feasibility and cost of 
data collection; 
-time-lag between 
input (research) and 
output (result) 
-possible 
underestimation of 

real impact 
-attribution issue 
-use of models and 
assumptions 



i.e. guidelines). The potential benefits of a research 
activity on population's health or its socio-economic sta- 
tus were more rarely addressed by the literature as they 
are, obviously, less directly linked and more complex to 
assess. In other words, these categories, with their respec- 
tive indicators, span into a gradient going from surrogate 
but easy to be measured outcomes, (i.e. bibliometric and 
citational data) to demanding but relevant outcomes (i.e. 
morbidity, quality of life). Bibliometric indicators (num- 
ber of publications, impact factor, citation indexes, etc.) 
were a case in point here. They were widely considered, 
reported, and to some extent, accepted due to the fact 
that they are easy to measure and outputs that can be 
straightforwardly attributed to a specific research activity. 
Only the studies adopting an econometric viewpoint [18] 
or evaluating a specific research area, such as primary 
care [27,28] or health system effectiveness [29] did not 
quote (or quoted with less emphasis) bibliometric 
indicators. 

Discussion 

Main findings 

This overview of reviews shows that the assessment of 
the impact of, or benefits from, health research is an 
issue of growing interest, mainly in those countries (UK, 
Canada, Australia, USA) that invest more in research. 
Research in this area focuses on three broad areas: 
i) theoretical frameworks and models aiming at asses- 
sing research impact with respect to multidimensional 
and integrated categories; ii) methodological approaches 
to the evaluation exercise; and iii) development of valid 
and reliable indicators and metrics. 

A common and key feature of most of the used mod- 
els is the multidimensional conceptualization and cate- 
gorization of research impact. Different impact aspects 
are connected and integrated using a variety of theoreti- 
cal approaches (i.e. Logic model for the Payback frame- 
work). Assessment of research impact that consider 
more than one category are indeed valued for their abil- 
ity to capture multifaceted processes. 



Several empirical approaches have been used to 
practically assess research impact: desk analysis, includ- 
ing bibliometrics, peer reviews, interviews, ad hoc case 
studies. The latter seems a reliable methodology: case 
study implies an explicit and a priori choice to start and 
conduct an evaluation exercise with specific aims and 
features. However, they can be at risk of "conceptualiza- 
tion bias" and "reporting bias" especially when they are 
designed or carried out retrospectively. Finally, feasibility 
and costs of case studies are also a major barrier to their 
conduct and subsequent use. In general, the methodol- 
ogy should be as flexible and adaptable as possible to 
many assessment questions, viewpoints, settings, and 
type of research and should guarantee the quality of 
collected data. 

The lack of standard terminology, the multifaceted 
nature of the evaluation, and the heterogeneity of the 
empirical experiences make it hard to identify a pre- 
ferred model. The most cited impact dimensions are 
related to knowledge, public health and socio-eco- 
nomic advancements. The Payback model, [38-41] and 
its adaptation into the Canadian framework [20] 
emerged as the most frequently quoted. Both are based 
on explicit assumptions (positive and negative), have 
been applied to empirical evaluation, and produce 
transparent categories of impact, indicators, and limita- 
tions of the models. They can be considered compre- 
hensive as assure a global approach to the evaluation 
of biomedical and health research impact. The identifi- 
cation of appropriate indicators is a critical step in any 
impact assessment exercise, and assessing research 
impact makes no exception to this. Indicators can be 
defined as factors or variables that provide simple and 
reliable means to measure impact, changes to an inter- 
vention, or performance [42]. Ideally, a set of a few 
robust, valid, shared, transferable, comparable, and fea- 
sible indicators able to synthesize research impact 
should be developed for any assessment. As a matter 
of fact, the usefulness of indicators highly depends on 
the evaluation purposes and the level of aggregation of 
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the unit of analysis: for instance, citation indicators 
partially capture the impact on knowledge advance- 
ments as they only consider published literature, they 
are artificially skewed by journal's impact factor and 
can be misleading when applied to individuals. More- 
over, an indicator itself informs only on a single aspect 
of research impact, thus sets of indicators are always 
advisable. 

This overview highlights the methodological limita- 
tions of the studies carried out in this field, which are 
briefly summarized below. 

First, the vast majority of the studies were retrospec- 
tive, based on interviews to principal investigators or 
funders, and mainly focused to record the projects' 
achievements rather that their pitfalls and limitations. 
This could lead to several biases, such as selective 
recall or reporting of (positive) results. The second 
major limitation is linked to the "attribution", the pos- 
sibility to postulate a causal link between observed (or 
expected to be observed) changes and a specific 
research activity [19]. Another limitation is linked to 
the possibility of understanding what would have hap- 
pened without "that" research activity (counterfactual). 
As very rarely a "control" situation is available, the 
identification of baseline measures and context factors 
is important in understanding what any counterfactual 
might have looked like. Finally, it is not commonly 
appreciated that substantial time lags exist between 
research funding and measurement of outputs. An 
impact assessment should be planned choosing an 
appropriate time window, which highly depends on the 
considered type of research and dimension of the 
impact. 

Limitations of this overview 

The main limitations of this overview concern the 
study retrieval process and the definition of the elig- 
ibility criteria. We experienced several difficulties in 
planning the search strategy, all caused by the hetero- 
geneity of definition and the lack of a standard termi- 
nology to describe "research impact". Bearing this in 
mind, we adopted an approach capable of maximizing 
sensitivity rather than specificity, that is the application 
of broad inclusion criteria and the use of several 
sources of information, not only bibliographic data- 
bases. As expected, only 30% of the included publica- 
tions were found through the traditional biomedical 
databases (i.e. Medline). Beyond a high variability in 
the way of indexing these articles, this could be due to 
a limited interest in the publication of these evalua- 
tions, which were considered, at least in the last dec- 
ade, an administrative duty rather than a scientific 



activity. Thus many relevant studies were retrieved in 
the "grey literature" (i.e. funding agency's reports) and 
not in scientific journals. Even if we were not able to 
apply a systematic approach to website consultation, 
we believe this effort had increased the comprehensive- 
ness of our search. 

Another methodological limitation of this overview is 
that we did not estimate the level of publication bias 
and selective publication in this field. Finally, as our 
analysis include study up to 2009, we did not capture 
new important emerging approaches to impact assess- 
ment, such as the Research Excellence Framework (for- 
merly RAE) [43]. 

Conclusions 

The main message of this overview is that the evaluation 
of the research impact is as yet a heterogeneous, and 
evolving discipline. Multidimensional conceptual frame- 
works appear to be adequate as they take into account 
several aspects of impact and use advanced and analyti- 
cal approaches (i.e. epidemiologic, economic, and social) 
to their evaluation. It remains to be clarified how differ- 
ent impact dimensions should be valued and balanced 
by assessors to fit them to their specific purposes and 
contexts. Added values to the multidimensional 
approach are to pursue an explicit planning of the 
assessment exercise and to carried out the alongside the 
development of research programs, through monitoring 
and prospective data collection. 

This overview should be seen as a preliminary step 
toward a shared conceptual framework and taxonomy to 
assess research impact rather than an indication of an 
ultimate model, that probably appears unrealistic. 
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